FLEX
Section: User Commands (1)
Updated: November 1993
Index
Return to Main Contents
NAME
flex - fast lexical analyzer generator
SYNOPSIS
flex
[-bcdfhilnpstvwBFILTV78+ -C[aefFmr] -Pprefix -Sskeleton]
[filename ...]
DESCRIPTION
flex
is a tool for generating
scanners:
programs which recognized lexical patterns in text.
flex
reads
the given input files, or its standard input if no file names are given,
for a description of a scanner to generate. The description is in
the form of pairs
of regular expressions and C code, called
rules. flex
generates as output a C source file,
lex.yy.c,
which defines a routine
yylex().
This file is compiled and linked with the
-lfl
library to produce an executable. When the executable is run,
it analyzes its input for occurrences
of the regular expressions. Whenever it finds one, it executes
the corresponding C code.
For full documentation, see
flexdoc(1).
This manual entry is intended for use as a quick reference.
OPTIONS
flex
has the following options:
- -b
-
generate backing-up information to
lex.backup.
This is a list of scanner states which require backing up and the input
characters on which they do so. By adding rules one can remove
backing-up states. If all backing-up states are eliminated and
-Cf
or
-CF
is used, the generated scanner will run faster.
- -c
-
is a do-nothing, deprecated option included for POSIX compliance.
-
NOTE:
in previous releases of
flex
-c
specified table-compression options. This functionality is
now given by the
-C
flag. To ease the the impact of this change, when
flex
encounters
-c,
it currently issues a warning message and assumes that
-C
was desired instead. In the future this "promotion" of
-c
to
-C
will go away in the name of full POSIX compliance (unless
the POSIX meaning is removed first).
- -d
-
makes the generated scanner run in
debug
mode. Whenever a pattern is recognized and the global
yy_flex_debug
is non-zero (which is the default), the scanner will
write to
stderr
a line of the form:
--accepting rule at line 53 ("the matched text")
The line number refers to the location of the rule in the file
defining the scanner (i.e., the file that was fed to flex). Messages
are also generated when the scanner backs up, accepts the
default rule, reaches the end of its input buffer (or encounters
a NUL; the two look the same as far as the scanner's concerned),
or reaches an end-of-file.
- -f
-
specifies
fast scanner.
No table compression is done and stdio is bypassed.
The result is large but fast. This option is equivalent to
-Cfr
(see below).
- -h
-
generates a "help" summary of
flex's
options to
stderr
and then exits.
- -i
-
instructs
flex
to generate a
case-insensitive
scanner. The case of letters given in the
flex
input patterns will
be ignored, and tokens in the input will be matched regardless of case. The
matched text given in
yytext
will have the preserved case (i.e., it will not be folded).
- -l
-
turns on maximum compatibility with the original AT&T lex implementation,
at a considerable performance cost. This option is incompatible with
-+, -f, -F, -Cf,
or
-CF.
See
flexdoc(1)
for details.
- -n
-
is another do-nothing, deprecated option included only for
POSIX compliance.
- -p
-
generates a performance report to stderr. The report
consists of comments regarding features of the
flex
input file which will cause a loss of performance in the resulting scanner.
If you give the flag twice, you will also get comments regarding
features that lead to minor performance losses.
- -s
-
causes the
default rule
(that unmatched scanner input is echoed to
stdout)
to be suppressed. If the scanner encounters input that does not
match any of its rules, it aborts with an error.
- -t
-
instructs
flex
to write the scanner it generates to standard output instead
of
lex.yy.c.
- -v
-
specifies that
flex
should write to
stderr
a summary of statistics regarding the scanner it generates.
- -w
-
suppresses warning messages.
- -B
-
instructs
flex
to generate a
batch
scanner instead of an
interactive
scanner (see
-I
below). See
flexdoc(1)
for details. Scanners using
-Cf
or
-CF
compression options automatically specify this option, too.
- -F
-
specifies that the
fast
scanner table representation should be used (and stdio bypassed).
This representation is about as fast as the full table representation
(-f),
and for some sets of patterns will be considerably smaller (and for
others, larger). It cannot be used with the
-+
option. See
flexdoc(1)
for more details.
-
This option is equivalent to
-CFr
(see below).
- -I
-
instructs
flex
to generate an
interactive
scanner, that is, a scanner which stops immediately rather than
looking ahead if it knows
that the currently scanned text cannot be part of a longer rule's match.
This is the opposite of
batch
scanners (see
-B
above). See
flexdoc(1)
for details.
-
Note,
-I
cannot be used in conjunction with
full
or
fast tables,
i.e., the
-f, -F, -Cf,
or
-CF
flags. For other table compression options,
-I
is the default.
- -L
-
instructs
flex
not to generate
#line
directives in
lex.yy.c.
The default is to generate such directives so error
messages in the actions will be correctly
located with respect to the original
flex
input file, and not to
the fairly meaningless line numbers of
lex.yy.c.
- -T
-
makes
flex
run in
trace
mode. It will generate a lot of messages to
stderr
concerning
the form of the input and the resultant non-deterministic and deterministic
finite automata. This option is mostly for use in maintaining
flex.
- -V
-
prints the version number to
stderr
and exits.
- -7
-
instructs
flex
to generate a 7-bit scanner, which can save considerable table space,
especially when using
-Cf
or
-CF
(and, at most sites,
-7
is on by default for these options. To see if this is the case, use the
-v
verbose flag and check the flag summary it reports).
- -8
-
instructs
flex
to generate an 8-bit scanner. This is the default except for the
-Cf
and
-CF
compression options, for which the default is site-dependent, and
can be checked by inspecting the flag summary generated by the
-v
option.
- -+
-
specifies that you want flex to generate a C++
scanner class. See the section on Generating C++ Scanners in
flexdoc(1)
for details.
- -C[aefFmr]
-
controls the degree of table compression and scanner optimization.
-
-Ca
trade off larger tables in the generated scanner for faster performance
because the elements of the tables are better aligned for memory access
and computation. This option can double the size of the tables used by
your scanner.
-
-Ce
directs
flex
to construct
equivalence classes,
i.e., sets of characters
which have identical lexical properties.
Equivalence classes usually give
dramatic reductions in the final table/object file sizes (typically
a factor of 2-5) and are pretty cheap performance-wise (one array
look-up per character scanned).
-
-Cf
specifies that the
full
scanner tables should be generated -
flex
should not compress the
tables by taking advantages of similar transition functions for
different states.
-
-CF
specifies that the alternate fast scanner representation (described in
flexdoc(1))
should be used. This option cannot be used with
-+.
-
-Cm
directs
flex
to construct
meta-equivalence classes,
which are sets of equivalence classes (or characters, if equivalence
classes are not being used) that are commonly used together. Meta-equivalence
classes are often a big win when using compressed tables, but they
have a moderate performance impact (one or two "if" tests and one
array look-up per character scanned).
-
-Cr
causes the generated scanner to
bypass
using stdio for input. In general this option results in a minor
performance gain only worthwhile if used in conjunction with
-Cf
or
-CF.
It can cause surprising behavior if you use stdio yourself to
read from
yyin
prior to calling the scanner.
-
A lone
-C
specifies that the scanner tables should be compressed but neither
equivalence classes nor meta-equivalence classes should be used.
-
The options
-Cf
or
-CF
and
-Cm
do not make sense together - there is no opportunity for meta-equivalence
classes if the table is not being compressed. Otherwise the options
may be freely mixed.
-
The default setting is
-Cem,
which specifies that
flex
should generate equivalence classes
and meta-equivalence classes. This setting provides the highest
degree of table compression. You can trade off
faster-executing scanners at the cost of larger tables with
the following generally being true:
slowest & smallest
-Cem
-Cm
-Ce
-C
-C{f,F}e
-C{f,F}
-C{f,F}a
fastest & largest
-
-C
options are cumulative.
- -Pprefix
-
changes the default
yy
prefix used by
flex
to be
prefix
instead. See
flexdoc(1)
for a description of all the global variables and file names that
this affects.
- -Sskeleton_file
-
overrides the default skeleton file from which
flex
constructs its scanners. You'll never need this option unless you are doing
flex
maintenance or development.
SUMMARY OF FLEX REGULAR EXPRESSIONS
The patterns in the input are written using an extended set of regular
expressions. These are:
x match the character 'x'
. any character except newline
[xyz] a "character class"; in this case, the pattern
matches either an 'x', a 'y', or a 'z'
[abj-oZ] a "character class" with a range in it; matches
an 'a', a 'b', any letter from 'j' through 'o',
or a 'Z'
[^A-Z] a "negated character class", i.e., any character
but those in the class. In this case, any
character EXCEPT an uppercase letter.
[^A-Z\n] any character EXCEPT an uppercase letter or
a newline
r* zero or more r's, where r is any regular expression
r+ one or more r's
r? zero or one r's (that is, "an optional r")
r{2,5} anywhere from two to five r's
r{2,} two or more r's
r{4} exactly 4 r's
{name} the expansion of the "name" definition
(see above)
"[xyz]\"foo"
the literal string: [xyz]"foo
\X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
then the ANSI-C interpretation of \x.
Otherwise, a literal 'X' (used to escape
operators such as '*')
\123 the character with octal value 123
\x2a the character with hexadecimal value 2a
(r) match an r; parentheses are used to override
precedence (see below)
rs the regular expression r followed by the
regular expression s; called "concatenation"
r|s either an r or an s
r/s an r but only if it is followed by an s. The
s is not part of the matched text. This type
of pattern is called as "trailing context".
^r an r, but only at the beginning of a line
r$ an r, but only at the end of a line. Equivalent
to "r/\n".
<s>r an r, but only in start condition s (see
below for discussion of start conditions)
<s1,s2,s3>r
same, but in any of start conditions s1,
s2, or s3
<*>r an r in any start condition, even an exclusive one.
<<EOF>> an end-of-file
<s1,s2><<EOF>>
an end-of-file when in start condition s1 or s2
The regular expressions listed above are grouped according to
precedence, from highest precedence at the top to lowest at the bottom.
Those grouped together have equal precedence.
Some notes on patterns:
- -
-
Negated character classes
match newlines
unless "\n" (or an equivalent escape sequence) is one of the
characters explicitly present in the negated character class
(e.g., "[^A-Z\n]").
- -
-
A rule can have at most one instance of trailing context (the '/' operator
or the '$' operator). The start condition, '^', and "<<EOF>>" patterns
can only occur at the beginning of a pattern, and, as well as with '/' and '$',
cannot be grouped inside parentheses. The following are all illegal:
foo/bar$
foo|(bar$)
foo|^bar
<sc1>foo<sc2>bar
SUMMARY OF SPECIAL ACTIONS
In addition to arbitrary C code, the following can appear in actions:
- -
-
ECHO
copies yytext to the scanner's output.
- -
-
BEGIN
followed by the name of a start condition places the scanner in the
corresponding start condition.
- -
-
REJECT
directs the scanner to proceed on to the "second best" rule which matched the
input (or a prefix of the input).
yytext
and
yyleng
are set up appropriately. Note that
REJECT
is a particularly expensive feature in terms scanner performance;
if it is used in
any
of the scanner's actions it will slow down
all
of the scanner's matching. Furthermore,
REJECT
cannot be used with the
-f
or
-F
options.
-
Note also that unlike the other special actions,
REJECT
is a
branch;
code immediately following it in the action will
not
be executed.
- -
-
yymore()
tells the scanner that the next time it matches a rule, the corresponding
token should be
appended
onto the current value of
yytext
rather than replacing it.
- -
-
yyless(n)
returns all but the first
n
characters of the current token back to the input stream, where they
will be rescanned when the scanner looks for the next match.
yytext
and
yyleng
are adjusted appropriately (e.g.,
yyleng
will now be equal to
n
).
- -
-
unput(c)
puts the character
c
back onto the input stream. It will be the next character scanned.
- -
-
input()
reads the next character from the input stream (this routine is called
yyinput()
if the scanner is compiled using
C++).
- -
-
yyterminate()
can be used in lieu of a return statement in an action. It terminates
the scanner and returns a 0 to the scanner's caller, indicating "all done".
-
By default,
yyterminate()
is also called when an end-of-file is encountered. It is a macro and
may be redefined.
- -
-
YY_NEW_FILE
is an action available only in <<EOF>> rules. It means "Okay, I've
set up a new input file, continue scanning". It is no longer required;
you can just assign
yyin
to point to a new file in the <<EOF>> action.
- -
-
yy_create_buffer( file, size )
takes a
FILE
pointer and an integer
size.
It returns a YY_BUFFER_STATE
handle to a new input buffer large enough to accomodate
size
characters and associated with the given file. When in doubt, use
YY_BUF_SIZE
for the size.
- -
-
yy_switch_to_buffer( new_buffer )
switches the scanner's processing to scan for tokens from
the given buffer, which must be a YY_BUFFER_STATE.
- -
-
yy_delete_buffer( buffer )
deletes the given buffer.
VALUES AVAILABLE TO THE USER
- -
-
char *yytext
holds the text of the current token. It may be modified but not lengthened
(you cannot append characters to the end). Modifying the last character
may affect the activity of rules anchored using '^' during the next scan;
see
flexdoc(1)
for details.
-
If the special directive
%array
appears in the first section of the scanner description, then
yytext
is instead declared
char yytext[YYLMAX],
where
YYLMAX
is a macro definition that you can redefine in the first section
if you don't like the default value (generally 8KB). Using
%array
results in somewhat slower scanners, but the value of
yytext
becomes immune to calls to
input()
and
unput(),
which potentially destroy its value when
yytext
is a character pointer. The opposite of
%array
is
%pointer,
which is the default.
-
You cannot use
%array
when generating C++ scanner classes
(the
-+
flag).
- -
-
int yyleng
holds the length of the current token.
- -
-
FILE *yyin
is the file which by default
flex
reads from. It may be redefined but doing so only makes sense before
scanning begins or after an EOF has been encountered. Changing it in
the midst of scanning will have unexpected results since
flex
buffers its input; use
yyrestart()
instead.
Once scanning terminates because an end-of-file
has been seen,
you can assign
yyin
at the new input file and then call the scanner again to continue scanning.
- -
-
void yyrestart( FILE *new_file )
may be called to point
yyin
at the new input file. The switch-over to the new file is immediate
(any previously buffered-up input is lost). Note that calling
yyrestart()
with
yyin
as an argument thus throws away the current input buffer and continues
scanning the same input file.
- -
-
FILE *yyout
is the file to which
ECHO
actions are done. It can be reassigned by the user.
- -
-
YY_CURRENT_BUFFER
returns a
YY_BUFFER_STATE
handle to the current buffer.
- -
-
YY_START
returns an integer value corresponding to the current start
condition. You can subsequently use this value with
BEGIN
to return to that start condition.
MACROS AND FUNCTIONS YOU CAN REDEFINE
- -
-
YY_DECL
controls how the scanning routine is declared.
By default, it is "int yylex()", or, if prototypes are being
used, "int yylex(void)". This definition may be changed by redefining
the "YY_DECL" macro. Note that
if you give arguments to the scanning routine using a
K&R-style/non-prototyped function declaration, you must terminate
the definition with a semi-colon (;).
- -
-
The nature of how the scanner
gets its input can be controlled by redefining the
YY_INPUT
macro.
YY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)". Its
action is to place up to
max_size
characters in the character array
buf
and return in the integer variable
result
either the
number of characters read or the constant YY_NULL (0 on Unix systems)
to indicate EOF. The default YY_INPUT reads from the
global file-pointer "yyin".
A sample redefinition of YY_INPUT (in the definitions
section of the input file):
%{
#undef YY_INPUT
#define YY_INPUT(buf,result,max_size) \
{ \
int c = getchar(); \
result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
}
%}
- -
-
When the scanner receives an end-of-file indication from YY_INPUT,
it then checks the function
yywrap()
function. If
yywrap()
returns false (zero), then it is assumed that the
function has gone ahead and set up
yyin
to point to another input file, and scanning continues. If it returns
true (non-zero), then the scanner terminates, returning 0 to its
caller.
-
The default
yywrap()
always returns 1.
- -
-
YY_USER_ACTION
can be redefined to provide an action
which is always executed prior to the matched rule's action.
- -
-
The macro
YY_USER_INIT
may be redefined to provide an action which is always executed before
the first scan.
- -
-
In the generated scanner, the actions are all gathered in one large
switch statement and separated using
YY_BREAK,
which may be redefined. By default, it is simply a "break", to separate
each rule's action from the following rule's.
FILES
- -lfl
-
library with which to link scanners to obtain the default versions
of
yywrap()
and/or
main().
- lex.yy.c
-
generated scanner (called
lexyy.c
on some systems).
- lex.yy.cc
-
generated C++ scanner class, when using
-+.
- <FlexLexer.h>
-
header file defining the C++ scanner base class,
FlexLexer,
and its derived class,
yyFlexLexer.
- flex.skl
-
skeleton scanner. This file is only used when building flex, not when
flex executes.
- lex.backup
-
backing-up information for
-b
flag (called
lex.bck
on some systems).
SEE ALSO
flexdoc(1), lex(1), yacc(1), sed(1), awk(1).
M. E. Lesk and E. Schmidt,
LEX - Lexical Analyzer Generator
DIAGNOSTICS
reject_used_but_not_detected undefined
or
yymore_used_but_not_detected undefined -
These errors can occur at compile time. They indicate that the
scanner uses
REJECT
or
yymore()
but that
flex
failed to notice the fact, meaning that
flex
scanned the first two sections looking for occurrences of these actions
and failed to find any, but somehow you snuck some in (via a #include
file, for example). Make an explicit reference to the action in your
flex
input file. (Note that previously
flex
supported a
%used/%unused
mechanism for dealing with this problem; this feature is still supported
but now deprecated, and will go away soon unless the author hears from
people who can argue compellingly that they need it.)
flex scanner jammed -
a scanner compiled with
-s
has encountered an input string which wasn't matched by
any of its rules.
warning, rule cannot be matched
indicates that the given rule
cannot be matched because it follows other rules that will
always match the same text as it. See
flexdoc(1)
for an example.
warning,
-s
option given but default rule can be matched
means that it is possible (perhaps only in a particular start condition)
that the default rule (match any single character) is the only one
that will match a particular input. Since
scanner input buffer overflowed -
a scanner rule matched more text than the available dynamic memory.
token too large, exceeds YYLMAX -
your scanner uses
%array
and one of its rules matched a string longer than the
YYLMAX
constant (8K bytes by default). You can increase the value by
#define'ing
YYLMAX
in the definitions section of your
flex
input.
scanner requires -8 flag to
use the character 'x' -
Your scanner specification includes recognizing the 8-bit character
'x'
and you did not specify the -8 flag, and your scanner defaulted to 7-bit
because you used the
-Cf
or
-CF
table compression options.
flex scanner push-back overflow -
you used
unput()
to push back so much text that the scanner's buffer could not hold
both the pushed-back text and the current token in
yytext.
Ideally the scanner should dynamically resize the buffer in this case, but at
present it does not.
input buffer overflow, can't enlarge buffer because scanner uses REJECT -
the scanner was working on matching an extremely large token and needed
to expand the input buffer. This doesn't work with scanners that use
REJECT.
fatal flex scanner internal error--end of buffer missed -
This can occur in an scanner which is reentered after a long-jump
has jumped out (or over) the scanner's activation frame. Before
reentering the scanner, use:
yyrestart( yyin );
or use C++ scanner classes (the
-+
option), which are fully reentrant.
AUTHOR
Vern Paxson, with the help of many ideas and much inspiration from
Van Jacobson. Original version by Jef Poskanzer.
See flexdoc(1) for additional credits and the address to send comments to.
DEFICIENCIES / BUGS
Some trailing context
patterns cannot be properly matched and generate
warning messages ("dangerous trailing context"). These are
patterns where the ending of the
first part of the rule matches the beginning of the second
part, such as "zx*/xy*", where the 'x*' matches the 'x' at
the beginning of the trailing context. (Note that the POSIX draft
states that the text matched by such patterns is undefined.)
For some trailing context rules, parts which are actually fixed-length are
not recognized as such, leading to the abovementioned performance loss.
In particular, parts using '|' or {n} (such as "foo{3}") are always
considered variable-length.
Combining trailing context with the special '|' action can result in
fixed
trailing context being turned into the more expensive
variable
trailing context. For example, in the following:
%%
abc |
xyz/def
Use of
unput()
or
input()
invalidates yytext and yyleng, unless the
%array
directive
or the
-l
option has been used.
Use of unput() to push back more text than was matched can
result in the pushed-back text matching a beginning-of-line ('^')
rule even though it didn't come at the beginning of the line
(though this is rare!).
Pattern-matching of NUL's is substantially slower than matching other
characters.
Dynamic resizing of the input buffer is slow, as it entails rescanning
all the text matched so far by the current (generally huge) token.
flex
does not generate correct #line directives for code internal
to the scanner; thus, bugs in
flex.skl
yield bogus line numbers.
Due to both buffering of input and read-ahead, you cannot intermix
calls to <stdio.h> routines, such as, for example,
getchar(),
with
flex
rules and expect it to work. Call
input()
instead.
The total table entries listed by the
-v
flag excludes the number of table entries needed to determine
what rule has been matched. The number of entries is equal
to the number of DFA states if the scanner does not use
REJECT,
and somewhat greater than the number of states if it does.
REJECT
cannot be used with the
-f
or
-F
options.
The
flex
internal algorithms need documentation.
Index
- NAME
-
- SYNOPSIS
-
- DESCRIPTION
-
- OPTIONS
-
- SUMMARY OF FLEX REGULAR EXPRESSIONS
-
- SUMMARY OF SPECIAL ACTIONS
-
- VALUES AVAILABLE TO THE USER
-
- MACROS AND FUNCTIONS YOU CAN REDEFINE
-
- FILES
-
- SEE ALSO
-
- DIAGNOSTICS
-
- AUTHOR
-
- DEFICIENCIES / BUGS
-
This document was created by
man2html,
using the manual pages.
Time: 17:16:03 GMT, January 16, 2023